All the GPU memory will be notoriously filled up even if you designate one GPU device.
maximum fraction
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))
automatic growth
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)
visible GPU:
os.environ[“CUDA_VISIBLE_DEVICES”] = “0,1” # use python to set environment variables
use multiple GPUs
One typical to use mulitple GPU is to average gradients, please refer to the sample code.